Bound repeated failure and eviction telemetry - #2148

Problem and Context

An open circuit breaker rejects retry invocations locally, but each rejection can emit network-error telemetry. Hydration and size-enumeration layers emit additional errors for the same operation. Successful LRU evictions also emit one telemetry event per eviction.

These repeated events can exhaust telemetry capacity and obscure genuine HTTP failures. Error counts across these layers do not represent HTTP requests or unique failed hydrations.

Changes

• Classify local breaker rejections with a dedicated exception. Annotate standard and custom handlers without changes to callbacks, retries, cancellation, or functional results.

• Distinguish direct hydration-retrier rejection from companion errors. Direct rejections use Rejected; nested-download and virtualizer companions use Failed.

• Bound known repetitive events at the telemetry daemon listener. Keep one diagnostic sample per key per interval, with at most 64 keys and a separate overflow counter.

• Emit TelemetrySummary events with observed, sampled, suppressed, and overflow counts. Use 60-second intervals that flush on subsequent traffic, plus a disposal flush. Idle periods can delay summaries.

• Use TelemetrySchemaVersion=3 and AggregationResult for the aggregation outcome. Preserve legacy result fields and HRESULT values without case-only JSON key collisions.

• Exclude SHA, paths, request IDs, command IDs, and exception text from aggregation keys. Forward unfamiliar errors and actual HTTP/authentication retry failures unchanged.

• Omit gitCommandSessionId from cross-command summaries. Preserve command context on individual samples.

• Retain every repeated error in direct local logs. Downstream ETW consumers receive the bounded pipe stream, not the suppressed raw events.

• Report observed breaker states without changes to thresholds or probes. Require a recorded cooldown expiry before a CooldownElapsed observation, and skip inconsistent snapshots.

• Group breaker samples by PreviousState and destination AggregationResult. Add LastObservedAtUtc to summaries without changes to retry decisions.

• Batch eviction diagnostics outside cache locks. Keep successful removals and failed eviction attempts in separate counters.

• Add unit tests for concurrent counters, key dimensions, overflow, wire fields, first-observation timestamps, and weighted eviction outcomes. The eviction test verifies its representative snapshot and cross-thread cache access during telemetry.

• Add unit tests for direct and companion classifications, failure-triggered breaker states, requestor tracer propagation, and missing child sizes. These tests verify unchanged cancellation, failure callbacks, and command completion.

• Add unit tests that reconnect a real pipe and verify shared child-tracer limits and complete local diagnostics. A disposal test verifies summary delivery and legacy HRESULT preservation.

• Add unit tests that switch command sessions and separate initial Accepting from subsequent recovery. These tests verify bounded repeated transitions and latest timestamps.

Summary counters describe per-layer observations, not unique failures. ObservedCount already includes the diagnostic samples. Transition summaries retain their latest receipt timestamp, not a complete ordered history or proof of service-wide recovery.

Known gaps and deliberate decisions

• Consumer prerequisites — Release scorecards and error-rate queries must use observation counts from samples and summaries before producer rollout. Raw-row comparisons can understate candidate failures.
• Local eviction diagnostics — Each mutation retains one representative eviction snapshot and its exact count. Local logs omit later evicted commit identifiers.
• Low-volume telemetry — An annotated observation emits a sample and a summary, even if no duplicate occurs. Each traced process also reports its initial breaker state.
• Capacity overflow — Known annotated groups that exceed capacity contribute exact overflow counts without individual samples. Unknown unannotated errors retain their full diagnostics.